None
The library imports relevant to this notebook are aready taken care of by importing PTT.
NOTE: This notebook assumes that the pipeline version to be tested is already installed and its environment is activated.
To be able to run this notebook you need to install nptt.
If all goes well you will be able to import PTT.
# Create a temporary directory to hold notebook output, and change the working directory to that directory.
import shutil
import os
from tempfile import TemporaryDirectory
use_tempdir = True
if use_tempdir:
# Create temporary directory
data_dir = TemporaryDirectory()
# Save original directory
orig_dir = os.getcwd()
# Move to new directory
os.chdir(data_dir.name)
# For info, print out where the script is running
print("Running in {}".format(os.getcwd()))
Running in /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5
import os
if 'CRDS_CACHE_TYPE' in os.environ:
if os.environ['CRDS_CACHE_TYPE'] == 'local':
os.environ['CRDS_PATH'] = os.path.join(os.environ['HOME'], 'crds', 'cache')
elif os.path.isdir(os.environ['CRDS_CACHE_TYPE']):
os.environ['CRDS_PATH'] = os.environ['CRDS_CACHE_TYPE']
print('CRDS cache location: {}'.format(os.environ['CRDS_PATH']))
CRDS cache location: /grp/crds/cache
import warnings
import psutil
from astropy.io import fits
# Only print a DeprecationWarning the first time it shows up, not every time.
with warnings.catch_warnings():
warnings.simplefilter("once", category=DeprecationWarning)
import jwst
from jwst.pipeline.calwebb_detector1 import Detector1Pipeline
from jwst.pipeline.calwebb_spec2 import Spec2Pipeline
from jwst.assign_wcs.assign_wcs_step import AssignWcsStep
from jwst.msaflagopen.msaflagopen_step import MSAFlagOpenStep
from jwst.extract_2d.extract_2d_step import Extract2dStep
from jwst.srctype.srctype_step import SourceTypeStep
from jwst.wavecorr.wavecorr_step import WavecorrStep
from jwst.flatfield.flat_field_step import FlatFieldStep
# The latest version of NPTT is installed in the requirements text file at:
# /jwst_validation_notebooks/environment.yml
# import NPTT
import nirspec_pipe_testing_tool as nptt
# To get data from Artifactory
from ci_watson.artifactory_helpers import get_bigdata
# Print versions used for the pipeline and NPTT
pipeline_version = jwst.__version__
nptt_version = nptt.__version__
print("Using jwst pipeline version: ", pipeline_version)
print("Using NPTT version: ", nptt_version)
Using jwst pipeline version: 1.7.2 Using NPTT version: 2.0.1
The test is a direct comparison of the result of our implementation of the flat field step algorithm versus the pipeline's implementation, i.e.: difference = absolute( Flat_nirspec_implementation - Flat_pipeline)
We expect the absolute difference to be of the order of 1x10^-6. We set this threshold by assuming that the difference should yield computer precision 1x10^-7 numbers. We then relaxed one order of magnitude due to interpolation differences in the algorithms.
For the test to be considered PASSED, every single slit (for FS data), slitlet (for MOS data) or slice (for IFU data) in the input file has to pass. If there is any failure, the whole test will be considered as FAILED.
The code for this test for Fixed Slits (FS) can be obtained from: https://github.com/spacetelescope/nirspec_pipe_testing_tool/blob/master/nirspec_pipe_testing_tool/calwebb_spec2_pytests/auxiliary_code/flattest_fs.py. For Multi Object Spectroscopy (MOS), the code is in the same repository but is named flattest_mos.py, and for Integral Field Unit (IFU) data, the test is named flattest_ifu.py.
The input file is defined in the variable input_file (see section Testing Data Set and Variable Setup).
Step description: https://jwst-pipeline.readthedocs.io/en/latest/jwst/flatfield/main.html
Pipeline code: https://github.com/spacetelescope/jwst/tree/master/jwst/flatfield
If the test PASSED this means that all slits, slitlets, or slices individually passed the test. However, if ony one individual slit (for FS data), slitlet (for MOS data) or slice (for IFU data) test failed, the whole test will be reported as FAILED.
A short description and link to the page: https://outerspace.stsci.edu/display/JWSTCC/Vanilla+Spectral+Flat+Field+Correction
Acronymns used un this notebook:
pipeline: calibration pipeline
spec2: spectroscopic calibration pipeline level 2b
PTT: NIRSpec pipeline testing tool (https://github.com/spacetelescope/nirspec_pipe_testing_tool)
The pipeline can be run from the command line in two variants: full or per step.
Tu run the spec2 pipeline in full use the command:
$ strun jwst.pipeline.Spec2Pipeline jwtest_rate.fits
Tu only run the flat_field step, use the command:
$ strun jwst.flat_field.FlatFieldStep jwtest_extract_2d.fits
These options are also callable from a script with the testing environment active. The Python call for running the pipeline in full or by step are:
$\gt$ from jwst.pipeline.calwebb_spec2 import Spec2Pipeline
$\gt$ Spec2Pipeline.call(jwtest_rate.fits)
or
$\gt$ from jwst.flat_field.flat_field_step import FlatFieldStep
$\gt$ FlatFieldStep.call(jwtest_extract_2d.fits)
For the imaging pipeline the call would be as follows:
$\gt$ from jwst.pipeline.calwebb_image2 import Image2Pipeline
$\gt$ Image2Pipeline.call(jwtest_rate.fits)
NPTT can run the spec2 pipeline either in full or per step, as well as the imaging pipeline in full. In this notebook we will use NPTT to run the pipeline and the validation tests. To run NPTT, follow the directions in the corresponding repo page.
-> For each mode, the following variables will need to be set:
All testing data is from the CV3 campaign. We chose these files because this is our most complete data set, i.e. all modes and filter-grating combinations.
Data used was for testing:
testing_data = {
'ifu_g395h_f290lp':{
'uncal_file_nrs1': 'ifu_g395h_f290lp_nrs1_uncal.fits',
'uncal_file_nrs2': 'ifu_g395h_f290lp_nrs2_uncal.fits',
'sflat_nrs1': 'nirspec_IFU_sflat_G395H_OPAQUE_FLAT3_nrs1_f_01.01.fits',
'sflat_nrs2': 'nirspec_IFU_sflat_G395H_OPAQUE_FLAT3_nrs2_f_01.01.fits',
'fflat': 'nirspec_IFU_fflat_F290LP_01.01.fits',
'msa_shutter_config': None }
}
# define function to pull data from Artifactory
def get_artifactory_file(data_set_dict, detector):
"""This function creates a list with all the files needed per detector to run the test.
Args:
data_set_dict: dictionary, contains inputs for a specific mode and configuration
detector: string, either nrs1 or nrs2
Returns:
data: list, contains all files needed to run test
"""
files2obtain = ['uncal_file_nrs1', 'sflat_nrs1', 'fflat', 'msa_shutter_config']
data = []
for file in files2obtain:
data_file = None
try:
if '_nrs' in file and '2' in detector:
file = file.replace('_nrs1', '_nrs2')
data_file = get_bigdata('jwst_validation_notebooks',
'validation_data',
'nirspec_data',
data_set_dict[file])
except TypeError:
data.append(None)
continue
data.append(data_file)
return data
# set the D-flat path (used for all test data)
print('Getting D-Flats from Artifactory...')
dflat_nrs1 = get_bigdata('jwst_validation_notebooks',
'validation_data',
'nirspec_data',
'nirspec_dflat_nrs1_f_01.03.fits')
dflat_nrs2 = get_bigdata('jwst_validation_notebooks',
'validation_data',
'nirspec_data',
'nirspec_dflat_nrs2_f_01.03.fits')
print('Got D-flats')
# set NPTT switches for this test and other variables
writefile = False
save_figs = False
show_figs = True
results_dict = {}
detectors = ['nrs1', 'nrs2']
Getting D-Flats from Artifactory... Got D-flats
# Get data for IFU
for mode_config, data_set_dict in testing_data.items():
if 'ifu' not in mode_config:
continue
print('Starting to run pipeline and test for mode: ', mode_config)
for detector in detectors:
print('Testing files for detector: ', detector)
data = get_artifactory_file(data_set_dict, detector)
uncal_file, sflat, fflat, msa_shutter_config = data
print('Working with uncal_file: ', uncal_file)
uncal_basename = os.path.basename(uncal_file)
dflat = dflat_nrs1
if '2' in detector:
dflat = dflat_nrs2
# Make sure these keywords are properly set
filt = fits.getval(uncal_file, 'FILTER')
if 'OPAQUE' in filt or 'allslits' in uncal_basename.lower():
if 'clear' in uncal_basename.lower():
filt = 'CLEAR'
else:
l = uncal_basename.split("_")
for li in l:
if 'lp' in li.lower():
filt = li
break
fits.setval(uncal_file, 'FILTER', value=filt)
print('Filter = ', filt)
# Run the stage 1 pipeline
print('Running the detector1 pipeline...')
rate_object = Detector1Pipeline.call(uncal_file)
# Make sure the FXD_SLIT keyword is set correctly
try:
if 'full' in rate_object.meta.instrument.fixed_slit:
rate_object.meta.instrument.fixed_slit = 'NONE'
except TypeError:
print('FXD_SLIT keyword = ', rate_object.meta.instrument.fixed_slit)
# Run the stage 2 pipeline steps
print('\nRunning the spec2 pipeline...')
skip_file = False
try:
parameter_dict = {"flat_field": {"save_interpolated_flat": True},
"pathloss": {"skip": True},
"barshadow": {"skip": True},
"photom": {"skip": True},
"resample_spec": {"skip": True},
"cube_build": {"skip": True},
"extract_1d": {"skip": True}
}
flat_field_object = Spec2Pipeline.call(rate_object, steps=parameter_dict)
except:
#print("No open slits fall on detector ", det) # usually why assign_wcs crashes
print("* Spec2 pipeline CRASHED or exited with no output for detector ", detector)
print(" Skipping test for this file. \n")
skip_file = True
if not skip_file:
# accepted threshold difference with respect to benchmark files
threshold_diff = 9.999e-5
if 'prism' in uncal_basename.lower():
threshold_diff = 9.999e-3
# Run the validation test
%matplotlib inline
interpolated_flat = os.path.basename(uncal_file).replace('uncal', 'interpolatedflat')
print('Running flat field test for IFU...')
result, result_msg, log_msgs = nptt.calwebb_spec2_pytests.auxiliary_code.flattest_ifu.flattest(
flat_field_object,
dflat_path=dflat,
sflat_path=sflat,
fflat_path=fflat,
writefile=writefile,
mk_all_slices_plt=False,
show_figs=show_figs,
save_figs=save_figs,
interpolated_flat=interpolated_flat,
threshold_diff=threshold_diff,
debug=False)
else:
result, result_msg = 'skipped', 'skipped'
# Did the test passed
print("Did flat_field for ", mode_config, " validation test passed? ", result_msg, "\n\n")
rd = {uncal_basename: result}
results_dict.update(rd)
# close all open files
psutil.Process().open_files()
closing_files = []
for fd in psutil.Process().open_files():
if data_dir.name in fd.path:
closing_files.append(fd)
for fd in closing_files:
try:
print('Closing file: ', fd, '\n\n')
open(fd.fd).close()
except:
print('File already closed: ', fd, '\n\n')
Starting to run pipeline and test for mode: ifu_g395h_f290lp Testing files for detector: nrs1 Working with uncal_file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/ifu_g395h_f290lp_nrs1_uncal.fits Filter = F290LP Running the detector1 pipeline...
2022-10-05 22:03:26,644 - stpipe - INFO - PARS-DETECTOR1PIPELINE parameters found: /grp/crds/cache/references/jwst/jwst_nirspec_pars-detector1pipeline_0004.asdf
2022-10-05 22:03:26,669 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created.
2022-10-05 22:03:26,670 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created.
2022-10-05 22:03:26,672 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created.
2022-10-05 22:03:26,673 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created.
2022-10-05 22:03:26,674 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created.
2022-10-05 22:03:26,675 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created.
2022-10-05 22:03:26,676 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created.
2022-10-05 22:03:26,677 - stpipe.Detector1Pipeline.rscd - INFO - RscdStep instance created.
2022-10-05 22:03:26,678 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created.
2022-10-05 22:03:26,679 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created.
2022-10-05 22:03:26,680 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created.
2022-10-05 22:03:26,681 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2022-10-05 22:03:26,683 - stpipe.Detector1Pipeline.reset - INFO - ResetStep instance created.
2022-10-05 22:03:26,684 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created.
2022-10-05 22:03:26,685 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created.
2022-10-05 22:03:26,686 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created.
2022-10-05 22:03:26,687 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created.
2022-10-05 22:03:26,806 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/ifu_g395h_f290lp_nrs1_uncal.fits',).
2022-10-05 22:03:26,816 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_calibrated_ramp': False, 'steps': {'group_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dq_init': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'saturation': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}, 'ipc': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'superbias': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'refpix': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}, 'rscd': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}, 'firstframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'lastframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'linearity': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}, 'reset': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'persistence': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'input_trapsfilled': '', 'flag_pers_cutoff': 40.0, 'save_persistence': False, 'save_trapsfilled': True}, 'jump': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 4.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}, 'ramp_fit': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}, 'gain_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}}}
2022-10-05 22:03:27,206 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'ifu_g395h_f290lp_nrs1_uncal.fits' reftypes = ['dark', 'gain', 'linearity', 'mask', 'persat', 'readnoise', 'refpix', 'reset', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars']
2022-10-05 22:03:27,222 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_dark_0088.fits'.
2022-10-05 22:03:27,223 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_gain_0023.fits'.
2022-10-05 22:03:27,225 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_linearity_0022.fits'.
2022-10-05 22:03:27,227 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_mask_0010.fits'.
2022-10-05 22:03:27,229 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is 'N/A'.
2022-10-05 22:03:27,229 - stpipe.Detector1Pipeline - INFO - Prefetch for READNOISE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_readnoise_0018.fits'.
2022-10-05 22:03:27,231 - stpipe.Detector1Pipeline - INFO - Prefetch for REFPIX reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_refpix_0019.fits'.
2022-10-05 22:03:27,232 - stpipe.Detector1Pipeline - INFO - Prefetch for RESET reference file is 'N/A'.
2022-10-05 22:03:27,232 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is 'N/A'.
2022-10-05 22:03:27,233 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_saturation_0027.fits'.
2022-10-05 22:03:27,234 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_superbias_0117.fits'.
2022-10-05 22:03:27,236 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is 'N/A'.
2022-10-05 22:03:27,236 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is 'N/A'.
2022-10-05 22:03:27,237 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ...
2022-10-05 22:03:28,092 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:03:28,093 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:03:28,602 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES and FRMDIVSR are equal; correction not needed
2022-10-05 22:03:28,603 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped
2022-10-05 22:03:28,605 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done
2022-10-05 22:03:28,725 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:03:28,727 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:03:28,749 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /grp/crds/cache/references/jwst/jwst_nirspec_mask_0010.fits
2022-10-05 22:03:30,040 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done
2022-10-05 22:03:30,203 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:03:30,205 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'n_pix_grow_sat': 1}
2022-10-05 22:03:30,230 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /grp/crds/cache/references/jwst/jwst_nirspec_saturation_0027.fits
2022-10-05 22:03:35,409 - stpipe.Detector1Pipeline.saturation - INFO - Detected 218077 saturated pixels
2022-10-05 22:03:35,555 - stpipe.Detector1Pipeline.saturation - INFO - Detected 0 A/D floor pixels
2022-10-05 22:03:35,582 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done
2022-10-05 22:03:35,705 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:03:35,707 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:03:35,707 - stpipe.Detector1Pipeline.ipc - INFO - Step skipped.
2022-10-05 22:03:35,710 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done
2022-10-05 22:03:35,819 - stpipe.Detector1Pipeline.superbias - INFO - Step superbias running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:03:35,820 - stpipe.Detector1Pipeline.superbias - INFO - Step superbias parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:03:35,844 - stpipe.Detector1Pipeline.superbias - INFO - Using SUPERBIAS reference file /grp/crds/cache/references/jwst/jwst_nirspec_superbias_0117.fits
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/stdatamodels/validate.py:38: ValidationWarning: While validating meta.exposure.readpatt the following error occurred:
'ALLIRS2' is not one of ['ACQ1', 'ACQ2', 'BRIGHT1', 'BRIGHT2', 'DEEP2', 'DEEP8', 'FAST', 'FASTGRPAVG', 'FASTGRPAVG8', 'FASTGRPAVG16', 'FASTGRPAVG32', 'FASTGRPAVG64', 'FASTR1', 'FASTR100', 'FGS', 'FGS60', 'FGS8370', 'FGS840', 'FGSRAPID', 'FINEGUIDE', 'ID', 'MEDIUM2', 'MEDIUM8', 'NIS', 'NISRAPID', 'NRS', 'NRSIRS2', 'NRSN16R4', 'NRSN32R8', 'NRSN8R2', 'NRSRAPID', 'NRSIRS2RAPID', 'NRSRAPIDD1', 'NRSRAPIDD2', 'NRSRAPIDD6', 'NRSSLOW', 'RAPID', 'SHALLOW2', 'SHALLOW4', 'SLOW', 'SLOWR1', 'TRACK', 'ANY', 'N/A']
Failed validating 'enum' in schema:
OrderedDict([('title', 'Readout pattern'),
('type', 'string'),
('enum',
['ACQ1',
'ACQ2',
'BRIGHT1',
'BRIGHT2',
'DEEP2',
'DEEP8',
'FAST',
'FASTGRPAVG',
'FASTGRPAVG8',
'FASTGRPAVG16',
'FASTGRPAVG32',
'FASTGRPAVG64',
'FASTR1',
'FASTR100',
'FGS',
'FGS60',
'FGS8370',
'FGS840',
'FGSRAPID',
'FINEGUIDE',
'ID',
'MEDIUM2',
'MEDIUM8',
'NIS',
'NISRAPID',
'NRS',
'NRSIRS2',
'NRSN16R4',
'NRSN32R8',
'NRSN8R2',
'NRSRAPID',
'NRSIRS2RAPID',
'NRSRAPIDD1',
'NRSRAPIDD2',
'NRSRAPIDD6',
'NRSSLOW',
'RAPID',
'SHALLOW2',
'SHALLOW4',
'SLOW',
'SLOWR1',
'TRACK',
'ANY',
'N/A']),
('fits_ke ...
warnings.warn(errmsg, ValidationWarning)
2022-10-05 22:03:38,428 - stpipe.Detector1Pipeline.superbias - INFO - Step superbias done
2022-10-05 22:03:38,550 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:03:38,552 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}
2022-10-05 22:03:38,575 - stpipe.Detector1Pipeline.refpix - INFO - Using refpix reference file: /grp/crds/cache/references/jwst/jwst_nirspec_refpix_0019.fits
2022-10-05 22:03:40,443 - stpipe.Detector1Pipeline.refpix - INFO - Working on integration 1
2022-10-05 22:04:55,533 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix done
2022-10-05 22:04:55,779 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:04:55,781 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:04:55,812 - stpipe.Detector1Pipeline.linearity - INFO - Using Linearity reference file /grp/crds/cache/references/jwst/jwst_nirspec_linearity_0022.fits
2022-10-05 22:04:56,616 - stpipe.Detector1Pipeline.linearity - WARNING - Keyword BAD_LIN_CORR does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-05 22:04:58,358 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity done
2022-10-05 22:04:58,484 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:04:58,485 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'dark_output': None}
2022-10-05 22:04:58,792 - stpipe.Detector1Pipeline.dark_current - INFO - Using DARK reference file /grp/crds/cache/references/jwst/jwst_nirspec_dark_0088.fits
2022-10-05 22:05:28,921 - stpipe.Detector1Pipeline.dark_current - INFO - Science data nints=1, ngroups=20, nframes=1, groupgap=0
2022-10-05 22:05:28,922 - stpipe.Detector1Pipeline.dark_current - INFO - Dark data nints=1, ngroups=65, nframes=1, groupgap=0
2022-10-05 22:05:30,488 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current done
2022-10-05 22:05:30,711 - stpipe.Detector1Pipeline.jump - INFO - Step jump running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:05:30,713 - stpipe.Detector1Pipeline.jump - INFO - Step jump parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'rejection_threshold': 4.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}
2022-10-05 22:05:30,861 - stpipe.Detector1Pipeline.jump - INFO - CR rejection threshold = 4 sigma
2022-10-05 22:05:31,027 - stpipe.Detector1Pipeline.jump - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_nirspec_gain_0023.fits
2022-10-05 22:05:31,569 - stpipe.Detector1Pipeline.jump - INFO - Using READNOISE reference file: /grp/crds/cache/references/jwst/jwst_nirspec_readnoise_0018.fits
2022-10-05 22:05:32,570 - stpipe.Detector1Pipeline.jump - INFO - Executing two-point difference method
2022-10-05 22:05:32,735 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 1:
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/stcal/jump/twopoint_difference.py:154: RuntimeWarning: All-NaN slice encountered
max_ratio = np.nanmax(ratio, axis=0)
2022-10-05 22:05:40,394 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 64946 pixels with at least one CR from five or more groups.
2022-10-05 22:05:40,396 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 442 pixels with at least one CR from four groups.
2022-10-05 22:05:40,396 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 437 pixels with at least one CR from three groups.
2022-10-05 22:05:59,441 - stpipe.Detector1Pipeline.jump - INFO - Total elapsed time = 26.8695 sec
2022-10-05 22:05:59,448 - stpipe.Detector1Pipeline.jump - INFO - The execution time in seconds: 28.587135
2022-10-05 22:05:59,454 - stpipe.Detector1Pipeline.jump - INFO - Step jump done
2022-10-05 22:05:59,726 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:05:59,728 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}
2022-10-05 22:06:00,110 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using READNOISE reference file: /grp/crds/cache/references/jwst/jwst_nirspec_readnoise_0018.fits
2022-10-05 22:06:00,111 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_nirspec_gain_0023.fits
2022-10-05 22:06:00,210 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using algorithm = ols
2022-10-05 22:06:00,211 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using weighting = optimal
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/stcal/ramp_fitting/ols_fit.py:1089: RuntimeWarning: invalid value encountered in multiply
var_p4[num_int, :, :, :] *= (segs_4[num_int, :, :, :] > 0)
2022-10-05 22:08:51,229 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of groups per integration: 20
2022-10-05 22:08:51,230 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of integrations: 1
2022-10-05 22:08:51,439 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit done
2022-10-05 22:08:51,622 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:08:51,624 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scale', 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:08:51,686 - stpipe.Detector1Pipeline.gain_scale - INFO - Rescaling by 1.0
2022-10-05 22:08:51,703 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-05 22:08:51,815 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<CubeModel(1, 2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:08:51,816 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scaleints', 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:08:51,880 - stpipe.Detector1Pipeline.gain_scale - INFO - Rescaling by 1.0
2022-10-05 22:08:51,898 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-05 22:08:51,898 - stpipe.Detector1Pipeline - INFO - ... ending calwebb_detector1
2022-10-05 22:08:51,899 - stpipe.Detector1Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-05 22:08:51,899 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline done
2022-10-05 22:08:51,944 - stpipe.Spec2Pipeline - INFO - Spec2Pipeline instance created.
2022-10-05 22:08:51,945 - stpipe.Spec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-10-05 22:08:51,946 - stpipe.Spec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-10-05 22:08:51,947 - stpipe.Spec2Pipeline.imprint_subtract - INFO - ImprintStep instance created.
2022-10-05 22:08:51,948 - stpipe.Spec2Pipeline.msa_flagging - INFO - MSAFlagOpenStep instance created.
2022-10-05 22:08:51,950 - stpipe.Spec2Pipeline.extract_2d - INFO - Extract2dStep instance created.
2022-10-05 22:08:51,953 - stpipe.Spec2Pipeline.master_background_mos - INFO - MasterBackgroundMosStep instance created.
2022-10-05 22:08:51,954 - stpipe.Spec2Pipeline.master_background_mos.flat_field - INFO - FlatFieldStep instance created.
2022-10-05 22:08:51,955 - stpipe.Spec2Pipeline.master_background_mos.pathloss - INFO - PathLossStep instance created.
2022-10-05 22:08:51,956 - stpipe.Spec2Pipeline.master_background_mos.barshadow - INFO - BarShadowStep instance created.
2022-10-05 22:08:51,957 - stpipe.Spec2Pipeline.master_background_mos.photom - INFO - PhotomStep instance created.
2022-10-05 22:08:51,958 - stpipe.Spec2Pipeline.wavecorr - INFO - WavecorrStep instance created.
2022-10-05 22:08:51,959 - stpipe.Spec2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-10-05 22:08:51,960 - stpipe.Spec2Pipeline.srctype - INFO - SourceTypeStep instance created.
2022-10-05 22:08:51,962 - stpipe.Spec2Pipeline.straylight - INFO - StraylightStep instance created.
2022-10-05 22:08:51,963 - stpipe.Spec2Pipeline.fringe - INFO - FringeStep instance created.
2022-10-05 22:08:51,964 - stpipe.Spec2Pipeline.pathloss - INFO - PathLossStep instance created.
2022-10-05 22:08:51,965 - stpipe.Spec2Pipeline.barshadow - INFO - BarShadowStep instance created.
2022-10-05 22:08:51,966 - stpipe.Spec2Pipeline.wfss_contam - INFO - WfssContamStep instance created.
2022-10-05 22:08:51,968 - stpipe.Spec2Pipeline.photom - INFO - PhotomStep instance created.
2022-10-05 22:08:51,970 - stpipe.Spec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created.
2022-10-05 22:08:51,971 - stpipe.Spec2Pipeline.cube_build - INFO - CubeBuildStep instance created.
2022-10-05 22:08:51,972 - stpipe.Spec2Pipeline.extract_1d - INFO - Extract1dStep instance created.
2022-10-05 22:08:52,086 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:08:52,100 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'fail_on_exception': True, 'save_wfss_esec': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'imprint_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'msa_flagging': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'extract_2d': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'slit_name': None, 'extract_orders': None, 'grism_objects': None, 'tsgrism_extract_height': None, 'wfss_extract_half_height': 5, 'wfss_mmag_extract': None, 'wfss_nbright': 1000}, 'master_background_mos': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'force_subtract': False, 'save_background': False, 'user_background': None, 'inverse': False, 'steps': {'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'pathloss': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'barshadow': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}}}, 'wavecorr': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': True, 'user_supplied_flat': None, 'inverse': False}, 'srctype': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'source_type': None}, 'straylight': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'fringe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'pathloss': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'barshadow': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'wfss_contam': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_simulated_image': False, 'save_contam_images': False, 'maximum_cores': 'none'}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample_spec': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}, 'cube_build': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'channel': 'all', 'band': 'all', 'grating': 'all', 'filter': 'all', 'output_type': 'band', 'scale1': 0.0, 'scale2': 0.0, 'scalew': 0.0, 'weighting': 'drizzle', 'coord_system': 'skyalign', 'rois': 0.0, 'roiw': 0.0, 'weight_power': 2.0, 'wavemin': None, 'wavemax': None, 'single': False, 'skip_dqflagging': False}, 'extract_1d': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'smoothing_length': None, 'bkg_fit': None, 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'center_xy': None, 'apply_apcorr': True, 'soss_threshold': 0.01, 'soss_n_os': 2, 'soss_transform': None, 'soss_tikfac': None, 'soss_width': 40.0, 'soss_bad_pix': 'model', 'soss_modelname': None}}}
2022-10-05 22:08:52,110 - stpipe.Spec2Pipeline - INFO - Prefetching reference files for dataset: 'ifu_g395h_f290lp_nrs1_uncal.fits' reftypes = ['camera', 'collimator', 'dflat', 'disperser', 'distortion', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'fringe', 'ifufore', 'ifupost', 'ifuslicer', 'mrsxartcorr', 'msa', 'msaoper', 'ote', 'regions', 'sflat', 'specwcs', 'wavecorr', 'wavelengthrange', 'wfssbkg']
Running the spec2 pipeline...
2022-10-05 22:08:52,131 - stpipe.Spec2Pipeline - INFO - Prefetch for CAMERA reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_camera_0004.asdf'.
2022-10-05 22:08:52,132 - stpipe.Spec2Pipeline - INFO - Prefetch for COLLIMATOR reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_collimator_0004.asdf'.
2022-10-05 22:08:52,134 - stpipe.Spec2Pipeline - INFO - Prefetch for DFLAT reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_dflat_0001.fits'.
2022-10-05 22:08:52,135 - stpipe.Spec2Pipeline - INFO - Prefetch for DISPERSER reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_disperser_0035.asdf'.
2022-10-05 22:08:52,137 - stpipe.Spec2Pipeline - INFO - Prefetch for DISTORTION reference file is 'N/A'.
2022-10-05 22:08:52,138 - stpipe.Spec2Pipeline - INFO - Prefetch for FFLAT reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_fflat_0010.fits'.
2022-10-05 22:08:52,139 - stpipe.Spec2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'.
2022-10-05 22:08:52,140 - stpipe.Spec2Pipeline - INFO - Prefetch for FLAT reference file is 'N/A'.
2022-10-05 22:08:52,140 - stpipe.Spec2Pipeline - INFO - Prefetch for FORE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_fore_0022.asdf'.
2022-10-05 22:08:52,141 - stpipe.Spec2Pipeline - INFO - Prefetch for FPA reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_fpa_0005.asdf'.
2022-10-05 22:08:52,144 - stpipe.Spec2Pipeline - INFO - Prefetch for FRINGE reference file is 'N/A'.
2022-10-05 22:08:52,144 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUFORE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_ifufore_0003.asdf'.
2022-10-05 22:08:52,146 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUPOST reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_ifupost_0004.asdf'.
2022-10-05 22:08:52,147 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUSLICER reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_ifuslicer_0003.asdf'.
2022-10-05 22:08:52,148 - stpipe.Spec2Pipeline - INFO - Prefetch for MRSXARTCORR reference file is 'N/A'.
2022-10-05 22:08:52,149 - stpipe.Spec2Pipeline - INFO - Prefetch for MSA reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_msa_0005.asdf'.
2022-10-05 22:08:52,151 - stpipe.Spec2Pipeline - INFO - Prefetch for MSAOPER reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_msaoper_0001.json'.
2022-10-05 22:08:52,152 - stpipe.Spec2Pipeline - INFO - Prefetch for OTE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_ote_0005.asdf'.
2022-10-05 22:08:52,155 - stpipe.Spec2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-10-05 22:08:52,155 - stpipe.Spec2Pipeline - INFO - Prefetch for SFLAT reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_sflat_0015.fits'.
2022-10-05 22:08:52,156 - stpipe.Spec2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-10-05 22:08:52,157 - stpipe.Spec2Pipeline - INFO - Prefetch for WAVECORR reference file is 'N/A'.
2022-10-05 22:08:52,157 - stpipe.Spec2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_wavelengthrange_0005.asdf'.
2022-10-05 22:08:52,158 - stpipe.Spec2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-10-05 22:08:52,159 - stpipe.Spec2Pipeline - INFO - Starting calwebb_spec2 ...
2022-10-05 22:08:52,217 - stpipe.Spec2Pipeline - INFO - Processing product ifu_g395h_f290lp_nrs1_uncal
2022-10-05 22:08:52,218 - stpipe.Spec2Pipeline - INFO - Working on input <IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits> ...
2022-10-05 22:08:52,340 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:08:52,341 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-10-05 22:08:52,546 - stpipe.Spec2Pipeline.assign_wcs - INFO - gwa_ytilt is 0.1448970586061478 deg
2022-10-05 22:08:52,547 - stpipe.Spec2Pipeline.assign_wcs - INFO - gwa_xtilt is 0.3232757747173309 deg
2022-10-05 22:08:52,548 - stpipe.Spec2Pipeline.assign_wcs - INFO - theta_y correction: -3.766823889842878e-13 deg
2022-10-05 22:08:52,549 - stpipe.Spec2Pipeline.assign_wcs - INFO - theta_x correction: 0.0 deg
2022-10-05 22:08:56,024 - stpipe.Spec2Pipeline.assign_wcs - INFO - Created a NIRSPEC nrs_ifu pipeline with references {'distortion': None, 'filteroffset': None, 'specwcs': None, 'regions': None, 'wavelengthrange': '/grp/crds/cache/references/jwst/jwst_nirspec_wavelengthrange_0005.asdf', 'camera': '/grp/crds/cache/references/jwst/jwst_nirspec_camera_0004.asdf', 'collimator': '/grp/crds/cache/references/jwst/jwst_nirspec_collimator_0004.asdf', 'disperser': '/grp/crds/cache/references/jwst/jwst_nirspec_disperser_0035.asdf', 'fore': '/grp/crds/cache/references/jwst/jwst_nirspec_fore_0022.asdf', 'fpa': '/grp/crds/cache/references/jwst/jwst_nirspec_fpa_0005.asdf', 'msa': '/grp/crds/cache/references/jwst/jwst_nirspec_msa_0005.asdf', 'ote': '/grp/crds/cache/references/jwst/jwst_nirspec_ote_0005.asdf', 'ifupost': '/grp/crds/cache/references/jwst/jwst_nirspec_ifupost_0004.asdf', 'ifufore': '/grp/crds/cache/references/jwst/jwst_nirspec_ifufore_0003.asdf', 'ifuslicer': '/grp/crds/cache/references/jwst/jwst_nirspec_ifuslicer_0003.asdf'}
2022-10-05 22:09:04,102 - stpipe.Spec2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 156.176999392 -45.687618156 156.178775679 -45.687618156 156.178775679 -45.686331563 156.176999392 -45.686331563
2022-10-05 22:09:04,105 - stpipe.Spec2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-10-05 22:09:04,119 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-10-05 22:09:04,432 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step bkg_subtract running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>, []).
2022-10-05 22:09:04,433 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step bkg_subtract parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}
2022-10-05 22:09:04,434 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step skipped.
2022-10-05 22:09:04,436 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step bkg_subtract done
2022-10-05 22:09:04,619 - stpipe.Spec2Pipeline.imprint_subtract - INFO - Step imprint_subtract running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>, []).
2022-10-05 22:09:04,620 - stpipe.Spec2Pipeline.imprint_subtract - INFO - Step imprint_subtract parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-05 22:09:04,621 - stpipe.Spec2Pipeline.imprint_subtract - INFO - Step skipped.
2022-10-05 22:09:04,624 - stpipe.Spec2Pipeline.imprint_subtract - INFO - Step imprint_subtract done
2022-10-05 22:09:04,806 - stpipe.Spec2Pipeline.msa_flagging - INFO - Step msa_flagging running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:09:04,808 - stpipe.Spec2Pipeline.msa_flagging - INFO - Step msa_flagging parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-05 22:09:04,829 - stpipe.Spec2Pipeline.msa_flagging - INFO - Using reference file /grp/crds/cache/references/jwst/jwst_nirspec_msaoper_0001.json
2022-10-05 22:09:04,831 - stpipe.JwstStep - INFO - JwstStep instance created.
2022-10-05 22:09:05,013 - stpipe.Spec2Pipeline.msa_flagging - INFO - gwa_ytilt is 0.1448970586061478 deg
2022-10-05 22:09:05,014 - stpipe.Spec2Pipeline.msa_flagging - INFO - gwa_xtilt is 0.3232757747173309 deg
2022-10-05 22:09:05,015 - stpipe.Spec2Pipeline.msa_flagging - INFO - theta_y correction: -3.766823889842878e-13 deg
2022-10-05 22:09:05,016 - stpipe.Spec2Pipeline.msa_flagging - INFO - theta_x correction: 0.0 deg
2022-10-05 22:09:05,033 - stpipe.Spec2Pipeline.msa_flagging - INFO - SPORDER= -1, wrange=[2.87e-06, 5.27e-06]
2022-10-05 22:09:05,228 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 5 open slits in quadrant 1
2022-10-05 22:09:05,279 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 3 open slits in quadrant 2
2022-10-05 22:09:05,306 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 9 open slits in quadrant 3
2022-10-05 22:09:05,384 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 3 open slits in quadrant 4
2022-10-05 22:09:05,411 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 0 open slits in quadrant 5
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/bounding_box.py:83: RuntimeWarning: Invalid interval: upper bound 2047.5 is strictly less than lower bound 2525.1239286219193.
warnings.warn(f"Invalid interval: upper bound {upper} "
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/bounding_box.py:83: RuntimeWarning: Invalid interval: upper bound 2047.5 is strictly less than lower bound 2239.8443211081017.
warnings.warn(f"Invalid interval: upper bound {upper} "
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/bounding_box.py:83: RuntimeWarning: Invalid interval: upper bound 2047.5 is strictly less than lower bound 2382.5735098584228.
warnings.warn(f"Invalid interval: upper bound {upper} "
2022-10-05 22:09:22,286 - stpipe.Spec2Pipeline.msa_flagging - INFO - Step msa_flagging done
2022-10-05 22:09:23,034 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:09:23,036 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'source_type': None}
2022-10-05 22:09:23,047 - stpipe.Spec2Pipeline.srctype - INFO - Input EXP_TYPE is NRS_IFU
2022-10-05 22:09:23,048 - stpipe.Spec2Pipeline.srctype - INFO - Input SRCTYAPT = UNKNOWN
2022-10-05 22:09:23,048 - stpipe.Spec2Pipeline.srctype - INFO - Input source type is unknown; setting default SRCTYPE = EXTENDED
2022-10-05 22:09:23,051 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype done
2022-10-05 22:09:23,404 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:09:23,406 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': True, 'user_supplied_flat': None, 'inverse': False}
2022-10-05 22:16:56,239 - stpipe.Spec2Pipeline.flat_field - INFO - Saved model in ifu_g395h_f290lp_nrs1_interpolatedflat.fits
2022-10-05 22:16:56,241 - stpipe.Spec2Pipeline.flat_field - INFO - Interpolated flat written to "ifu_g395h_f290lp_nrs1_interpolatedflat.fits".
2022-10-05 22:16:56,248 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field done
2022-10-05 22:16:57,439 - stpipe.Spec2Pipeline.straylight - INFO - Step straylight running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:16:57,441 - stpipe.Spec2Pipeline.straylight - INFO - Step straylight parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-05 22:16:57,443 - stpipe.Spec2Pipeline.straylight - INFO - Step skipped.
2022-10-05 22:16:57,446 - stpipe.Spec2Pipeline.straylight - INFO - Step straylight done
2022-10-05 22:16:57,763 - stpipe.Spec2Pipeline.fringe - INFO - Step fringe running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:16:57,765 - stpipe.Spec2Pipeline.fringe - INFO - Step fringe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-05 22:16:57,766 - stpipe.Spec2Pipeline.fringe - INFO - Step skipped.
2022-10-05 22:16:57,769 - stpipe.Spec2Pipeline.fringe - INFO - Step fringe done
2022-10-05 22:16:58,086 - stpipe.Spec2Pipeline.pathloss - INFO - Step pathloss running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:16:58,088 - stpipe.Spec2Pipeline.pathloss - INFO - Step pathloss parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-05 22:16:58,089 - stpipe.Spec2Pipeline.pathloss - INFO - Step skipped.
2022-10-05 22:16:58,092 - stpipe.Spec2Pipeline.pathloss - INFO - Step pathloss done
2022-10-05 22:16:58,410 - stpipe.Spec2Pipeline.barshadow - INFO - Step barshadow running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:16:58,411 - stpipe.Spec2Pipeline.barshadow - INFO - Step barshadow parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-05 22:16:58,412 - stpipe.Spec2Pipeline.barshadow - INFO - Step skipped.
2022-10-05 22:16:58,415 - stpipe.Spec2Pipeline.barshadow - INFO - Step barshadow done
2022-10-05 22:16:58,734 - stpipe.Spec2Pipeline.photom - INFO - Step photom running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_uncal.fits>,).
2022-10-05 22:16:58,735 - stpipe.Spec2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-05 22:16:58,736 - stpipe.Spec2Pipeline.photom - INFO - Step skipped.
2022-10-05 22:16:58,739 - stpipe.Spec2Pipeline.photom - INFO - Step photom done
2022-10-05 22:16:59,091 - stpipe.Spec2Pipeline.cube_build - INFO - Step cube_build running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_cal.fits>,).
2022-10-05 22:16:59,093 - stpipe.Spec2Pipeline.cube_build - INFO - Step cube_build parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'channel': 'all', 'band': 'all', 'grating': 'all', 'filter': 'all', 'output_type': 'multi', 'scale1': 0.0, 'scale2': 0.0, 'scalew': 0.0, 'weighting': 'drizzle', 'coord_system': 'skyalign', 'rois': 0.0, 'roiw': 0.0, 'weight_power': 2.0, 'wavemin': None, 'wavemax': None, 'single': False, 'skip_dqflagging': True}
2022-10-05 22:16:59,094 - stpipe.Spec2Pipeline.cube_build - INFO - Step skipped.
2022-10-05 22:16:59,097 - stpipe.Spec2Pipeline.cube_build - INFO - Step cube_build done
2022-10-05 22:16:59,341 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs1_cal.fits>,).
2022-10-05 22:16:59,343 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'x1d', 'search_output_file': True, 'input_dir': '', 'smoothing_length': None, 'bkg_fit': None, 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'center_xy': None, 'apply_apcorr': True, 'soss_threshold': 0.01, 'soss_n_os': 2, 'soss_transform': None, 'soss_tikfac': None, 'soss_width': 40.0, 'soss_bad_pix': 'model', 'soss_modelname': None}
2022-10-05 22:16:59,344 - stpipe.Spec2Pipeline.extract_1d - INFO - Step skipped.
2022-10-05 22:16:59,346 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d done
2022-10-05 22:16:59,347 - stpipe.Spec2Pipeline - INFO - Finished processing product ifu_g395h_f290lp_nrs1_uncal
2022-10-05 22:16:59,347 - stpipe.Spec2Pipeline - INFO - Ending calwebb_spec2
2022-10-05 22:16:59,348 - stpipe.Spec2Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-05 22:16:59,348 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline done
Running flat field test for IFU...
flat_field_file --> Grating:G395H Filter:F290LP LAMP:REF
Getting and reading the D-, S-, and F-flats for this specific IFU configuration...
Using D-flat: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_dflat_nrs1_f_01.03.fits
Using S-flat: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_IFU_sflat_G395H_OPAQUE_FLAT3_nrs1_f_01.01.fits
Using F-flat: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_IFU_fflat_F290LP_01.01.fits
Now looping through the slices, this may take some time...
Working with slice: 00
Subwindow origin: px0=275 py0=782
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 00
Absolute Flat Difference : mean = 2.451e-07 median = 2.961e-07 stdev = 9.459e-05
Maximum AbsoluteFlat Difference = 6.543e-04
Minimum AbsoluteFlat Difference = -8.418e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 25%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 01
Subwindow origin: px0=312 py0=1198
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 01
Absolute Flat Difference : mean = 6.821e-08 median = 2.047e-07 stdev = 1.043e-04
Maximum AbsoluteFlat Difference = 1.462e-03
Minimum AbsoluteFlat Difference = -9.074e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 02
Subwindow origin: px0=271 py0=732
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 02
Absolute Flat Difference : mean = 9.632e-08 median = 4.646e-07 stdev = 9.722e-05
Maximum AbsoluteFlat Difference = 1.025e-03
Minimum AbsoluteFlat Difference = -8.078e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 26%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 03
Subwindow origin: px0=317 py0=1247
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 03
Absolute Flat Difference : mean = -7.211e-09 median = 6.240e-07 stdev = 1.029e-04
Maximum AbsoluteFlat Difference = 7.566e-04
Minimum AbsoluteFlat Difference = -8.425e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 28%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 04
Subwindow origin: px0=266 py0=683
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 04
Absolute Flat Difference : mean = 2.326e-07 median = 5.106e-07 stdev = 1.008e-04
Maximum AbsoluteFlat Difference = 7.220e-03
Minimum AbsoluteFlat Difference = -8.518e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 26%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 05
Subwindow origin: px0=321 py0=1297
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 05
Absolute Flat Difference : mean = -1.699e-07 median = 3.482e-07 stdev = 1.025e-04
Maximum AbsoluteFlat Difference = 9.310e-04
Minimum AbsoluteFlat Difference = -1.075e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 28%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 06
Subwindow origin: px0=262 py0=634
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 06
Absolute Flat Difference : mean = -4.773e-08 median = 1.812e-07 stdev = 9.605e-05
Maximum AbsoluteFlat Difference = 1.507e-03
Minimum AbsoluteFlat Difference = -7.785e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 26%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 07
Subwindow origin: px0=326 py0=1346
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 07
Absolute Flat Difference : mean = -1.528e-07 median = 7.333e-08 stdev = 1.030e-04
Maximum AbsoluteFlat Difference = 7.676e-04
Minimum AbsoluteFlat Difference = -1.083e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 08
Subwindow origin: px0=258 py0=585
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 08
Absolute Flat Difference : mean = 1.205e-07 median = 5.618e-07 stdev = 9.567e-05
Maximum AbsoluteFlat Difference = 6.572e-04
Minimum AbsoluteFlat Difference = -2.243e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 26%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 09
Subwindow origin: px0=331 py0=1395
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 09
Absolute Flat Difference : mean = 3.251e-07 median = 2.518e-07 stdev = 1.032e-04
Maximum AbsoluteFlat Difference = 9.500e-04
Minimum AbsoluteFlat Difference = -7.979e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 10
Subwindow origin: px0=254 py0=535
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 10
Absolute Flat Difference : mean = -1.037e-07 median = 2.018e-07 stdev = 9.510e-05
Maximum AbsoluteFlat Difference = 6.764e-04
Minimum AbsoluteFlat Difference = -1.602e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 25%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 11
Subwindow origin: px0=336 py0=1443
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 11
Absolute Flat Difference : mean = -4.067e-09 median = 1.756e-07 stdev = 1.035e-04
Maximum AbsoluteFlat Difference = 9.278e-04
Minimum AbsoluteFlat Difference = -7.630e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 12
Subwindow origin: px0=250 py0=486
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 12
Absolute Flat Difference : mean = -1.508e-08 median = 1.683e-07 stdev = 9.473e-05
Maximum AbsoluteFlat Difference = 8.629e-04
Minimum AbsoluteFlat Difference = -7.682e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 25%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 13
Subwindow origin: px0=340 py0=1492
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 13
Absolute Flat Difference : mean = 2.747e-08 median = 7.277e-07 stdev = 1.051e-04
Maximum AbsoluteFlat Difference = 7.780e-04
Minimum AbsoluteFlat Difference = -1.011e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 14
Subwindow origin: px0=246 py0=437
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 14
Absolute Flat Difference : mean = -8.771e-08 median = 1.375e-07 stdev = 9.393e-05
Maximum AbsoluteFlat Difference = 1.175e-03
Minimum AbsoluteFlat Difference = -8.226e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 25%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 15
Subwindow origin: px0=345 py0=1541
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 15
Absolute Flat Difference : mean = 6.037e-08 median = 2.856e-07 stdev = 1.036e-04
Maximum AbsoluteFlat Difference = 7.866e-04
Minimum AbsoluteFlat Difference = -1.065e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 16
Subwindow origin: px0=242 py0=388
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 16
Absolute Flat Difference : mean = 1.999e-07 median = 4.584e-07 stdev = 9.374e-05
Maximum AbsoluteFlat Difference = 7.020e-04
Minimum AbsoluteFlat Difference = -6.467e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 25%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 17
Subwindow origin: px0=350 py0=1590
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 17
Absolute Flat Difference : mean = 9.924e-08 median = 7.185e-08 stdev = 1.042e-04
Maximum AbsoluteFlat Difference = 8.168e-04
Minimum AbsoluteFlat Difference = -9.450e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 18
Subwindow origin: px0=238 py0=338
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 18
Absolute Flat Difference : mean = -8.228e-08 median = 3.453e-07 stdev = 9.320e-05
Maximum AbsoluteFlat Difference = 7.553e-04
Minimum AbsoluteFlat Difference = -8.363e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 25%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 19
Subwindow origin: px0=355 py0=1639
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 19
Absolute Flat Difference : mean = 1.096e-07 median = 2.492e-07 stdev = 1.047e-04
Maximum AbsoluteFlat Difference = 8.936e-04
Minimum AbsoluteFlat Difference = -1.133e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 20
Subwindow origin: px0=235 py0=289
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 20
Absolute Flat Difference : mean = 8.732e-08 median = 4.784e-07 stdev = 9.306e-05
Maximum AbsoluteFlat Difference = 7.746e-04
Minimum AbsoluteFlat Difference = -8.747e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 24%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 21
Subwindow origin: px0=360 py0=1688
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 21
Absolute Flat Difference : mean = -1.734e-08 median = 2.882e-07 stdev = 1.060e-04
Maximum AbsoluteFlat Difference = 8.270e-04
Minimum AbsoluteFlat Difference = -1.754e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 30%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 22
Subwindow origin: px0=231 py0=239
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 22
Absolute Flat Difference : mean = 1.374e-08 median = 3.011e-07 stdev = 9.305e-05
Maximum AbsoluteFlat Difference = 7.602e-04
Minimum AbsoluteFlat Difference = -9.745e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 24%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 23
Subwindow origin: px0=365 py0=1737
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 23
Absolute Flat Difference : mean = -2.059e-07 median = -9.524e-08 stdev = 1.059e-04
Maximum AbsoluteFlat Difference = 8.519e-04
Minimum AbsoluteFlat Difference = -9.368e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 29%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 24
Subwindow origin: px0=227 py0=190
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 24
Absolute Flat Difference : mean = 1.764e-08 median = 3.316e-07 stdev = 9.550e-05
Maximum AbsoluteFlat Difference = 1.096e-03
Minimum AbsoluteFlat Difference = -9.782e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 25%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 25
Subwindow origin: px0=370 py0=1785
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 25
Absolute Flat Difference : mean = 4.924e-08 median = 3.957e-07 stdev = 1.069e-04
Maximum AbsoluteFlat Difference = 8.774e-04
Minimum AbsoluteFlat Difference = -8.857e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 30%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 26
Subwindow origin: px0=223 py0=141
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 26
Absolute Flat Difference : mean = -9.450e-08 median = 4.652e-07 stdev = 9.876e-05
Maximum AbsoluteFlat Difference = 1.342e-03
Minimum AbsoluteFlat Difference = -2.304e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 24%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 27
Subwindow origin: px0=375 py0=1834
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 27
Absolute Flat Difference : mean = -2.183e-08 median = 7.055e-07 stdev = 1.078e-04
Maximum AbsoluteFlat Difference = 7.290e-04
Minimum AbsoluteFlat Difference = -9.310e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 30%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 28
Subwindow origin: px0=219 py0=91
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 28
Absolute Flat Difference : mean = 9.470e-07 median = 3.703e-07 stdev = 1.486e-04
Maximum AbsoluteFlat Difference = 1.010e-02
Minimum AbsoluteFlat Difference = -8.962e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 20%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 29
Subwindow origin: px0=380 py0=1883
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 29
Absolute Flat Difference : mean = -1.895e-08 median = -2.349e-07 stdev = 1.082e-04
Maximum AbsoluteFlat Difference = 8.523e-04
Minimum AbsoluteFlat Difference = -1.091e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 31%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
*** Final result for flat_field test will be reported as PASSED ***
('* Script flattest_ifu.py script took ', '39.95554734071096 minutes to finish.')
Did flat_field for ifu_g395h_f290lp validation test passed? All slices PASSED flat_field test.
Closing file: popenfile(path='/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/ifu_g395h_f290lp_nrs1_interpolatedflat.fits', fd=56, position=16804800, mode='r', flags=557056)
Closing file: popenfile(path='/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_dflat_nrs1_f_01.03.fits', fd=57, position=680480640, mode='r', flags=557056)
Closing file: popenfile(path='/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_IFU_sflat_G395H_OPAQUE_FLAT3_nrs1_f_01.01.fits', fd=58, position=50500800, mode='r', flags=557056)
Closing file: popenfile(path='/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_IFU_fflat_F290LP_01.01.fits', fd=59, position=86400, mode='r', flags=557056)
Testing files for detector: nrs2
Working with uncal_file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/ifu_g395h_f290lp_nrs2_uncal.fits
Filter = F290LP
Running the detector1 pipeline...
2022-10-05 22:56:59,661 - stpipe - INFO - PARS-DETECTOR1PIPELINE parameters found: /grp/crds/cache/references/jwst/jwst_nirspec_pars-detector1pipeline_0004.asdf
2022-10-05 22:56:59,690 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created.
2022-10-05 22:56:59,692 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created.
2022-10-05 22:56:59,693 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created.
2022-10-05 22:56:59,694 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created.
2022-10-05 22:56:59,695 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created.
2022-10-05 22:56:59,696 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created.
2022-10-05 22:56:59,697 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created.
2022-10-05 22:56:59,698 - stpipe.Detector1Pipeline.rscd - INFO - RscdStep instance created.
2022-10-05 22:56:59,699 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created.
2022-10-05 22:56:59,701 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created.
2022-10-05 22:56:59,701 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created.
2022-10-05 22:56:59,702 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2022-10-05 22:56:59,703 - stpipe.Detector1Pipeline.reset - INFO - ResetStep instance created.
2022-10-05 22:56:59,704 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created.
2022-10-05 22:56:59,706 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created.
2022-10-05 22:56:59,707 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created.
2022-10-05 22:56:59,709 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created.
2022-10-05 22:57:00,529 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/ifu_g395h_f290lp_nrs2_uncal.fits',).
2022-10-05 22:57:00,540 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_calibrated_ramp': False, 'steps': {'group_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dq_init': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'saturation': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}, 'ipc': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'superbias': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'refpix': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}, 'rscd': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}, 'firstframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'lastframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'linearity': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}, 'reset': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'persistence': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'input_trapsfilled': '', 'flag_pers_cutoff': 40.0, 'save_persistence': False, 'save_trapsfilled': True}, 'jump': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 4.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}, 'ramp_fit': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}, 'gain_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}}}
2022-10-05 22:57:00,907 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'ifu_g395h_f290lp_nrs2_uncal.fits' reftypes = ['dark', 'gain', 'linearity', 'mask', 'persat', 'readnoise', 'refpix', 'reset', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars']
2022-10-05 22:57:00,922 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_dark_0085.fits'.
2022-10-05 22:57:00,924 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_gain_0020.fits'.
2022-10-05 22:57:00,926 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_linearity_0025.fits'.
2022-10-05 22:57:00,927 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_mask_0011.fits'.
2022-10-05 22:57:00,928 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is 'N/A'.
2022-10-05 22:57:00,928 - stpipe.Detector1Pipeline - INFO - Prefetch for READNOISE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_readnoise_0017.fits'.
2022-10-05 22:57:00,930 - stpipe.Detector1Pipeline - INFO - Prefetch for REFPIX reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_refpix_0018.fits'.
2022-10-05 22:57:00,931 - stpipe.Detector1Pipeline - INFO - Prefetch for RESET reference file is 'N/A'.
2022-10-05 22:57:00,932 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is 'N/A'.
2022-10-05 22:57:00,932 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_saturation_0026.fits'.
2022-10-05 22:57:00,934 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_superbias_0118.fits'.
2022-10-05 22:57:00,935 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is 'N/A'.
2022-10-05 22:57:00,936 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is 'N/A'.
2022-10-05 22:57:00,936 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ...
2022-10-05 22:57:01,844 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:57:01,846 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:57:02,331 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES and FRMDIVSR are equal; correction not needed
2022-10-05 22:57:02,331 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped
2022-10-05 22:57:02,334 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done
2022-10-05 22:57:02,545 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:57:02,546 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:57:02,568 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /grp/crds/cache/references/jwst/jwst_nirspec_mask_0011.fits
2022-10-05 22:57:03,384 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done
2022-10-05 22:57:03,612 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:57:03,613 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'n_pix_grow_sat': 1}
2022-10-05 22:57:03,635 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /grp/crds/cache/references/jwst/jwst_nirspec_saturation_0026.fits
2022-10-05 22:57:07,924 - stpipe.Detector1Pipeline.saturation - INFO - Detected 224463 saturated pixels
2022-10-05 22:57:08,066 - stpipe.Detector1Pipeline.saturation - INFO - Detected 0 A/D floor pixels
2022-10-05 22:57:08,088 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done
2022-10-05 22:57:08,290 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:57:08,292 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:57:08,293 - stpipe.Detector1Pipeline.ipc - INFO - Step skipped.
2022-10-05 22:57:08,295 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done
2022-10-05 22:57:08,504 - stpipe.Detector1Pipeline.superbias - INFO - Step superbias running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:57:08,506 - stpipe.Detector1Pipeline.superbias - INFO - Step superbias parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:57:08,529 - stpipe.Detector1Pipeline.superbias - INFO - Using SUPERBIAS reference file /grp/crds/cache/references/jwst/jwst_nirspec_superbias_0118.fits
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/stdatamodels/validate.py:38: ValidationWarning: While validating meta.exposure.readpatt the following error occurred:
'ALLIRS2' is not one of ['ACQ1', 'ACQ2', 'BRIGHT1', 'BRIGHT2', 'DEEP2', 'DEEP8', 'FAST', 'FASTGRPAVG', 'FASTGRPAVG8', 'FASTGRPAVG16', 'FASTGRPAVG32', 'FASTGRPAVG64', 'FASTR1', 'FASTR100', 'FGS', 'FGS60', 'FGS8370', 'FGS840', 'FGSRAPID', 'FINEGUIDE', 'ID', 'MEDIUM2', 'MEDIUM8', 'NIS', 'NISRAPID', 'NRS', 'NRSIRS2', 'NRSN16R4', 'NRSN32R8', 'NRSN8R2', 'NRSRAPID', 'NRSIRS2RAPID', 'NRSRAPIDD1', 'NRSRAPIDD2', 'NRSRAPIDD6', 'NRSSLOW', 'RAPID', 'SHALLOW2', 'SHALLOW4', 'SLOW', 'SLOWR1', 'TRACK', 'ANY', 'N/A']
Failed validating 'enum' in schema:
OrderedDict([('title', 'Readout pattern'),
('type', 'string'),
('enum',
['ACQ1',
'ACQ2',
'BRIGHT1',
'BRIGHT2',
'DEEP2',
'DEEP8',
'FAST',
'FASTGRPAVG',
'FASTGRPAVG8',
'FASTGRPAVG16',
'FASTGRPAVG32',
'FASTGRPAVG64',
'FASTR1',
'FASTR100',
'FGS',
'FGS60',
'FGS8370',
'FGS840',
'FGSRAPID',
'FINEGUIDE',
'ID',
'MEDIUM2',
'MEDIUM8',
'NIS',
'NISRAPID',
'NRS',
'NRSIRS2',
'NRSN16R4',
'NRSN32R8',
'NRSN8R2',
'NRSRAPID',
'NRSIRS2RAPID',
'NRSRAPIDD1',
'NRSRAPIDD2',
'NRSRAPIDD6',
'NRSSLOW',
'RAPID',
'SHALLOW2',
'SHALLOW4',
'SLOW',
'SLOWR1',
'TRACK',
'ANY',
'N/A']),
('fits_ke ...
warnings.warn(errmsg, ValidationWarning)
2022-10-05 22:57:11,985 - stpipe.Detector1Pipeline.superbias - INFO - Step superbias done
2022-10-05 22:57:12,210 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:57:12,211 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}
2022-10-05 22:57:12,234 - stpipe.Detector1Pipeline.refpix - INFO - Using refpix reference file: /grp/crds/cache/references/jwst/jwst_nirspec_refpix_0018.fits
2022-10-05 22:57:14,396 - stpipe.Detector1Pipeline.refpix - INFO - Working on integration 1
2022-10-05 22:58:23,023 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix done
2022-10-05 22:58:23,329 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:58:23,330 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 22:58:23,359 - stpipe.Detector1Pipeline.linearity - INFO - Using Linearity reference file /grp/crds/cache/references/jwst/jwst_nirspec_linearity_0025.fits
2022-10-05 22:58:25,143 - stpipe.Detector1Pipeline.linearity - WARNING - Keyword BAD_LIN_CORR does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-05 22:58:26,797 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity done
2022-10-05 22:58:27,006 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:58:27,007 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'dark_output': None}
2022-10-05 22:58:27,291 - stpipe.Detector1Pipeline.dark_current - INFO - Using DARK reference file /grp/crds/cache/references/jwst/jwst_nirspec_dark_0085.fits
2022-10-05 22:58:55,449 - stpipe.Detector1Pipeline.dark_current - INFO - Science data nints=1, ngroups=20, nframes=1, groupgap=0
2022-10-05 22:58:55,450 - stpipe.Detector1Pipeline.dark_current - INFO - Dark data nints=1, ngroups=65, nframes=1, groupgap=0
2022-10-05 22:58:56,978 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current done
2022-10-05 22:58:57,205 - stpipe.Detector1Pipeline.jump - INFO - Step jump running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:58:57,207 - stpipe.Detector1Pipeline.jump - INFO - Step jump parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'rejection_threshold': 4.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}
2022-10-05 22:58:57,350 - stpipe.Detector1Pipeline.jump - INFO - CR rejection threshold = 4 sigma
2022-10-05 22:58:57,501 - stpipe.Detector1Pipeline.jump - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_nirspec_gain_0020.fits
2022-10-05 22:58:58,079 - stpipe.Detector1Pipeline.jump - INFO - Using READNOISE reference file: /grp/crds/cache/references/jwst/jwst_nirspec_readnoise_0017.fits
2022-10-05 22:58:59,620 - stpipe.Detector1Pipeline.jump - INFO - Executing two-point difference method
2022-10-05 22:58:59,783 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 1:
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/stcal/jump/twopoint_difference.py:154: RuntimeWarning: All-NaN slice encountered
max_ratio = np.nanmax(ratio, axis=0)
2022-10-05 22:59:07,554 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 68624 pixels with at least one CR from five or more groups.
2022-10-05 22:59:07,555 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 408 pixels with at least one CR from four groups.
2022-10-05 22:59:07,556 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 36 pixels with at least one CR from three groups.
2022-10-05 22:59:32,816 - stpipe.Detector1Pipeline.jump - INFO - Total elapsed time = 33.1951 sec
2022-10-05 22:59:32,819 - stpipe.Detector1Pipeline.jump - INFO - The execution time in seconds: 35.468438
2022-10-05 22:59:32,824 - stpipe.Detector1Pipeline.jump - INFO - Step jump done
2022-10-05 22:59:33,033 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit running with args (<RampModel(1, 20, 3200, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 22:59:33,035 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}
2022-10-05 22:59:33,416 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using READNOISE reference file: /grp/crds/cache/references/jwst/jwst_nirspec_readnoise_0017.fits
2022-10-05 22:59:33,417 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_nirspec_gain_0020.fits
2022-10-05 22:59:33,506 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using algorithm = ols
2022-10-05 22:59:33,506 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using weighting = optimal
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/stcal/ramp_fitting/ols_fit.py:1089: RuntimeWarning: invalid value encountered in multiply
var_p4[num_int, :, :, :] *= (segs_4[num_int, :, :, :] > 0)
2022-10-05 23:02:18,739 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of groups per integration: 20
2022-10-05 23:02:18,740 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of integrations: 1
2022-10-05 23:02:18,925 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit done
2022-10-05 23:02:19,174 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:02:19,175 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scale', 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 23:02:19,231 - stpipe.Detector1Pipeline.gain_scale - INFO - Rescaling by 1.0
2022-10-05 23:02:19,247 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-05 23:02:19,452 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<CubeModel(1, 2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:02:19,453 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scaleints', 'search_output_file': True, 'input_dir': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5'}
2022-10-05 23:02:19,511 - stpipe.Detector1Pipeline.gain_scale - INFO - Rescaling by 1.0
2022-10-05 23:02:19,529 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-05 23:02:19,530 - stpipe.Detector1Pipeline - INFO - ... ending calwebb_detector1
2022-10-05 23:02:19,530 - stpipe.Detector1Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-05 23:02:19,531 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline done
2022-10-05 23:02:19,574 - stpipe.Spec2Pipeline - INFO - Spec2Pipeline instance created.
2022-10-05 23:02:19,576 - stpipe.Spec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-10-05 23:02:19,577 - stpipe.Spec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-10-05 23:02:19,578 - stpipe.Spec2Pipeline.imprint_subtract - INFO - ImprintStep instance created.
2022-10-05 23:02:19,579 - stpipe.Spec2Pipeline.msa_flagging - INFO - MSAFlagOpenStep instance created.
2022-10-05 23:02:19,581 - stpipe.Spec2Pipeline.extract_2d - INFO - Extract2dStep instance created.
2022-10-05 23:02:19,583 - stpipe.Spec2Pipeline.master_background_mos - INFO - MasterBackgroundMosStep instance created.
2022-10-05 23:02:19,584 - stpipe.Spec2Pipeline.master_background_mos.flat_field - INFO - FlatFieldStep instance created.
2022-10-05 23:02:19,586 - stpipe.Spec2Pipeline.master_background_mos.pathloss - INFO - PathLossStep instance created.
2022-10-05 23:02:19,587 - stpipe.Spec2Pipeline.master_background_mos.barshadow - INFO - BarShadowStep instance created.
2022-10-05 23:02:19,588 - stpipe.Spec2Pipeline.master_background_mos.photom - INFO - PhotomStep instance created.
2022-10-05 23:02:19,589 - stpipe.Spec2Pipeline.wavecorr - INFO - WavecorrStep instance created.
2022-10-05 23:02:19,590 - stpipe.Spec2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-10-05 23:02:19,590 - stpipe.Spec2Pipeline.srctype - INFO - SourceTypeStep instance created.
2022-10-05 23:02:19,592 - stpipe.Spec2Pipeline.straylight - INFO - StraylightStep instance created.
2022-10-05 23:02:19,593 - stpipe.Spec2Pipeline.fringe - INFO - FringeStep instance created.
2022-10-05 23:02:19,594 - stpipe.Spec2Pipeline.pathloss - INFO - PathLossStep instance created.
2022-10-05 23:02:19,595 - stpipe.Spec2Pipeline.barshadow - INFO - BarShadowStep instance created.
2022-10-05 23:02:19,596 - stpipe.Spec2Pipeline.wfss_contam - INFO - WfssContamStep instance created.
2022-10-05 23:02:19,597 - stpipe.Spec2Pipeline.photom - INFO - PhotomStep instance created.
2022-10-05 23:02:19,600 - stpipe.Spec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created.
2022-10-05 23:02:19,601 - stpipe.Spec2Pipeline.cube_build - INFO - CubeBuildStep instance created.
2022-10-05 23:02:19,603 - stpipe.Spec2Pipeline.extract_1d - INFO - Extract1dStep instance created.
2022-10-05 23:02:19,807 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
Running the spec2 pipeline...
2022-10-05 23:02:19,821 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'fail_on_exception': True, 'save_wfss_esec': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'imprint_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'msa_flagging': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'extract_2d': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'slit_name': None, 'extract_orders': None, 'grism_objects': None, 'tsgrism_extract_height': None, 'wfss_extract_half_height': 5, 'wfss_mmag_extract': None, 'wfss_nbright': 1000}, 'master_background_mos': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'force_subtract': False, 'save_background': False, 'user_background': None, 'inverse': False, 'steps': {'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'pathloss': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'barshadow': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}}}, 'wavecorr': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': True, 'user_supplied_flat': None, 'inverse': False}, 'srctype': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'source_type': None}, 'straylight': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'fringe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'pathloss': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'barshadow': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'wfss_contam': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_simulated_image': False, 'save_contam_images': False, 'maximum_cores': 'none'}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample_spec': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}, 'cube_build': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'channel': 'all', 'band': 'all', 'grating': 'all', 'filter': 'all', 'output_type': 'band', 'scale1': 0.0, 'scale2': 0.0, 'scalew': 0.0, 'weighting': 'drizzle', 'coord_system': 'skyalign', 'rois': 0.0, 'roiw': 0.0, 'weight_power': 2.0, 'wavemin': None, 'wavemax': None, 'single': False, 'skip_dqflagging': False}, 'extract_1d': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'smoothing_length': None, 'bkg_fit': None, 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'center_xy': None, 'apply_apcorr': True, 'soss_threshold': 0.01, 'soss_n_os': 2, 'soss_transform': None, 'soss_tikfac': None, 'soss_width': 40.0, 'soss_bad_pix': 'model', 'soss_modelname': None}}}
2022-10-05 23:02:19,830 - stpipe.Spec2Pipeline - INFO - Prefetching reference files for dataset: 'ifu_g395h_f290lp_nrs2_uncal.fits' reftypes = ['camera', 'collimator', 'dflat', 'disperser', 'distortion', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'fringe', 'ifufore', 'ifupost', 'ifuslicer', 'mrsxartcorr', 'msa', 'msaoper', 'ote', 'regions', 'sflat', 'specwcs', 'wavecorr', 'wavelengthrange', 'wfssbkg']
2022-10-05 23:02:19,858 - stpipe.Spec2Pipeline - INFO - Prefetch for CAMERA reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_camera_0004.asdf'.
2022-10-05 23:02:19,860 - stpipe.Spec2Pipeline - INFO - Prefetch for COLLIMATOR reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_collimator_0004.asdf'.
2022-10-05 23:02:19,861 - stpipe.Spec2Pipeline - INFO - Prefetch for DFLAT reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_dflat_0002.fits'.
2022-10-05 23:02:19,863 - stpipe.Spec2Pipeline - INFO - Prefetch for DISPERSER reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_disperser_0035.asdf'.
2022-10-05 23:02:19,864 - stpipe.Spec2Pipeline - INFO - Prefetch for DISTORTION reference file is 'N/A'.
2022-10-05 23:02:19,864 - stpipe.Spec2Pipeline - INFO - Prefetch for FFLAT reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_fflat_0010.fits'.
2022-10-05 23:02:19,866 - stpipe.Spec2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is 'N/A'.
2022-10-05 23:02:19,866 - stpipe.Spec2Pipeline - INFO - Prefetch for FLAT reference file is 'N/A'.
2022-10-05 23:02:19,867 - stpipe.Spec2Pipeline - INFO - Prefetch for FORE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_fore_0022.asdf'.
2022-10-05 23:02:19,868 - stpipe.Spec2Pipeline - INFO - Prefetch for FPA reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_fpa_0005.asdf'.
2022-10-05 23:02:19,869 - stpipe.Spec2Pipeline - INFO - Prefetch for FRINGE reference file is 'N/A'.
2022-10-05 23:02:19,870 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUFORE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_ifufore_0003.asdf'.
2022-10-05 23:02:19,871 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUPOST reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_ifupost_0004.asdf'.
2022-10-05 23:02:19,872 - stpipe.Spec2Pipeline - INFO - Prefetch for IFUSLICER reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_ifuslicer_0003.asdf'.
2022-10-05 23:02:19,874 - stpipe.Spec2Pipeline - INFO - Prefetch for MRSXARTCORR reference file is 'N/A'.
2022-10-05 23:02:19,874 - stpipe.Spec2Pipeline - INFO - Prefetch for MSA reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_msa_0005.asdf'.
2022-10-05 23:02:19,875 - stpipe.Spec2Pipeline - INFO - Prefetch for MSAOPER reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_msaoper_0001.json'.
2022-10-05 23:02:19,877 - stpipe.Spec2Pipeline - INFO - Prefetch for OTE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_ote_0005.asdf'.
2022-10-05 23:02:19,878 - stpipe.Spec2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-10-05 23:02:19,879 - stpipe.Spec2Pipeline - INFO - Prefetch for SFLAT reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_sflat_0016.fits'.
2022-10-05 23:02:19,880 - stpipe.Spec2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-10-05 23:02:19,880 - stpipe.Spec2Pipeline - INFO - Prefetch for WAVECORR reference file is 'N/A'.
2022-10-05 23:02:19,881 - stpipe.Spec2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is '/grp/crds/cache/references/jwst/jwst_nirspec_wavelengthrange_0005.asdf'.
2022-10-05 23:02:19,882 - stpipe.Spec2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-10-05 23:02:19,883 - stpipe.Spec2Pipeline - INFO - Starting calwebb_spec2 ...
2022-10-05 23:02:19,939 - stpipe.Spec2Pipeline - INFO - Processing product ifu_g395h_f290lp_nrs2_uncal
2022-10-05 23:02:19,940 - stpipe.Spec2Pipeline - INFO - Working on input <IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits> ...
2022-10-05 23:02:20,155 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:02:20,157 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-10-05 23:02:20,344 - stpipe.Spec2Pipeline.assign_wcs - INFO - gwa_ytilt is 0.1448970586061478 deg
2022-10-05 23:02:20,345 - stpipe.Spec2Pipeline.assign_wcs - INFO - gwa_xtilt is 0.3232757747173309 deg
2022-10-05 23:02:20,346 - stpipe.Spec2Pipeline.assign_wcs - INFO - theta_y correction: -3.766823889842878e-13 deg
2022-10-05 23:02:20,347 - stpipe.Spec2Pipeline.assign_wcs - INFO - theta_x correction: 0.0 deg
2022-10-05 23:02:23,514 - stpipe.Spec2Pipeline.assign_wcs - INFO - Created a NIRSPEC nrs_ifu pipeline with references {'distortion': None, 'filteroffset': None, 'specwcs': None, 'regions': None, 'wavelengthrange': '/grp/crds/cache/references/jwst/jwst_nirspec_wavelengthrange_0005.asdf', 'camera': '/grp/crds/cache/references/jwst/jwst_nirspec_camera_0004.asdf', 'collimator': '/grp/crds/cache/references/jwst/jwst_nirspec_collimator_0004.asdf', 'disperser': '/grp/crds/cache/references/jwst/jwst_nirspec_disperser_0035.asdf', 'fore': '/grp/crds/cache/references/jwst/jwst_nirspec_fore_0022.asdf', 'fpa': '/grp/crds/cache/references/jwst/jwst_nirspec_fpa_0005.asdf', 'msa': '/grp/crds/cache/references/jwst/jwst_nirspec_msa_0005.asdf', 'ote': '/grp/crds/cache/references/jwst/jwst_nirspec_ote_0005.asdf', 'ifupost': '/grp/crds/cache/references/jwst/jwst_nirspec_ifupost_0004.asdf', 'ifufore': '/grp/crds/cache/references/jwst/jwst_nirspec_ifufore_0003.asdf', 'ifuslicer': '/grp/crds/cache/references/jwst/jwst_nirspec_ifuslicer_0003.asdf'}
2022-10-05 23:02:30,849 - stpipe.Spec2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 156.176996289 -45.687617183 156.178772625 -45.687617183 156.178772625 -45.686330364 156.176996289 -45.686330364
2022-10-05 23:02:30,851 - stpipe.Spec2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-10-05 23:02:30,864 - stpipe.Spec2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-10-05 23:02:31,199 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step bkg_subtract running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>, []).
2022-10-05 23:02:31,201 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step bkg_subtract parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}
2022-10-05 23:02:31,201 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step skipped.
2022-10-05 23:02:31,204 - stpipe.Spec2Pipeline.bkg_subtract - INFO - Step bkg_subtract done
2022-10-05 23:02:31,469 - stpipe.Spec2Pipeline.imprint_subtract - INFO - Step imprint_subtract running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>, []).
2022-10-05 23:02:31,470 - stpipe.Spec2Pipeline.imprint_subtract - INFO - Step imprint_subtract parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-05 23:02:31,471 - stpipe.Spec2Pipeline.imprint_subtract - INFO - Step skipped.
2022-10-05 23:02:31,474 - stpipe.Spec2Pipeline.imprint_subtract - INFO - Step imprint_subtract done
2022-10-05 23:02:31,743 - stpipe.Spec2Pipeline.msa_flagging - INFO - Step msa_flagging running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:02:31,745 - stpipe.Spec2Pipeline.msa_flagging - INFO - Step msa_flagging parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-05 23:02:31,765 - stpipe.Spec2Pipeline.msa_flagging - INFO - Using reference file /grp/crds/cache/references/jwst/jwst_nirspec_msaoper_0001.json
2022-10-05 23:02:31,767 - stpipe.JwstStep - INFO - JwstStep instance created.
2022-10-05 23:02:31,962 - stpipe.Spec2Pipeline.msa_flagging - INFO - gwa_ytilt is 0.1448970586061478 deg
2022-10-05 23:02:31,963 - stpipe.Spec2Pipeline.msa_flagging - INFO - gwa_xtilt is 0.3232757747173309 deg
2022-10-05 23:02:31,963 - stpipe.Spec2Pipeline.msa_flagging - INFO - theta_y correction: -3.766823889842878e-13 deg
2022-10-05 23:02:31,965 - stpipe.Spec2Pipeline.msa_flagging - INFO - theta_x correction: 0.0 deg
2022-10-05 23:02:31,981 - stpipe.Spec2Pipeline.msa_flagging - INFO - SPORDER= -1, wrange=[2.87e-06, 5.27e-06]
2022-10-05 23:02:32,161 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 5 open slits in quadrant 1
2022-10-05 23:02:32,204 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 3 open slits in quadrant 2
2022-10-05 23:02:32,229 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 9 open slits in quadrant 3
2022-10-05 23:02:32,302 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 3 open slits in quadrant 4
2022-10-05 23:02:32,327 - stpipe.Spec2Pipeline.msa_flagging - INFO - There are 0 open slits in quadrant 5
2022-10-05 23:02:47,694 - stpipe.Spec2Pipeline.msa_flagging - INFO - Step msa_flagging done
2022-10-05 23:02:48,285 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:02:48,288 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'source_type': None}
2022-10-05 23:02:48,298 - stpipe.Spec2Pipeline.srctype - INFO - Input EXP_TYPE is NRS_IFU
2022-10-05 23:02:48,299 - stpipe.Spec2Pipeline.srctype - INFO - Input SRCTYAPT = UNKNOWN
2022-10-05 23:02:48,300 - stpipe.Spec2Pipeline.srctype - INFO - Input source type is unknown; setting default SRCTYPE = EXTENDED
2022-10-05 23:02:48,303 - stpipe.Spec2Pipeline.srctype - INFO - Step srctype done
2022-10-05 23:02:48,820 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:02:48,822 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': True, 'user_supplied_flat': None, 'inverse': False}
2022-10-05 23:10:23,734 - stpipe.Spec2Pipeline.flat_field - INFO - Saved model in ifu_g395h_f290lp_nrs2_interpolatedflat.fits
2022-10-05 23:10:23,736 - stpipe.Spec2Pipeline.flat_field - INFO - Interpolated flat written to "ifu_g395h_f290lp_nrs2_interpolatedflat.fits".
2022-10-05 23:10:23,742 - stpipe.Spec2Pipeline.flat_field - INFO - Step flat_field done
2022-10-05 23:10:25,564 - stpipe.Spec2Pipeline.straylight - INFO - Step straylight running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:10:25,567 - stpipe.Spec2Pipeline.straylight - INFO - Step straylight parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-05 23:10:25,568 - stpipe.Spec2Pipeline.straylight - INFO - Step skipped.
2022-10-05 23:10:25,570 - stpipe.Spec2Pipeline.straylight - INFO - Step straylight done
2022-10-05 23:10:26,291 - stpipe.Spec2Pipeline.fringe - INFO - Step fringe running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:10:26,293 - stpipe.Spec2Pipeline.fringe - INFO - Step fringe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-05 23:10:26,293 - stpipe.Spec2Pipeline.fringe - INFO - Step skipped.
2022-10-05 23:10:26,296 - stpipe.Spec2Pipeline.fringe - INFO - Step fringe done
2022-10-05 23:10:26,983 - stpipe.Spec2Pipeline.pathloss - INFO - Step pathloss running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:10:26,985 - stpipe.Spec2Pipeline.pathloss - INFO - Step pathloss parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-05 23:10:26,985 - stpipe.Spec2Pipeline.pathloss - INFO - Step skipped.
2022-10-05 23:10:26,988 - stpipe.Spec2Pipeline.pathloss - INFO - Step pathloss done
2022-10-05 23:10:27,674 - stpipe.Spec2Pipeline.barshadow - INFO - Step barshadow running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:10:27,676 - stpipe.Spec2Pipeline.barshadow - INFO - Step barshadow parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-05 23:10:27,677 - stpipe.Spec2Pipeline.barshadow - INFO - Step skipped.
2022-10-05 23:10:27,680 - stpipe.Spec2Pipeline.barshadow - INFO - Step barshadow done
2022-10-05 23:10:28,369 - stpipe.Spec2Pipeline.photom - INFO - Step photom running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_uncal.fits>,).
2022-10-05 23:10:28,371 - stpipe.Spec2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-05 23:10:28,371 - stpipe.Spec2Pipeline.photom - INFO - Step skipped.
2022-10-05 23:10:28,374 - stpipe.Spec2Pipeline.photom - INFO - Step photom done
2022-10-05 23:10:29,117 - stpipe.Spec2Pipeline.cube_build - INFO - Step cube_build running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_cal.fits>,).
2022-10-05 23:10:29,120 - stpipe.Spec2Pipeline.cube_build - INFO - Step cube_build parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'channel': 'all', 'band': 'all', 'grating': 'all', 'filter': 'all', 'output_type': 'multi', 'scale1': 0.0, 'scale2': 0.0, 'scalew': 0.0, 'weighting': 'drizzle', 'coord_system': 'skyalign', 'rois': 0.0, 'roiw': 0.0, 'weight_power': 2.0, 'wavemin': None, 'wavemax': None, 'single': False, 'skip_dqflagging': True}
2022-10-05 23:10:29,121 - stpipe.Spec2Pipeline.cube_build - INFO - Step skipped.
2022-10-05 23:10:29,124 - stpipe.Spec2Pipeline.cube_build - INFO - Step cube_build done
2022-10-05 23:10:29,690 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d running with args (<IFUImageModel(2048, 2048) from ifu_g395h_f290lp_nrs2_cal.fits>,).
2022-10-05 23:10:29,692 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': 'x1d', 'search_output_file': True, 'input_dir': '', 'smoothing_length': None, 'bkg_fit': None, 'bkg_order': None, 'bkg_sigma_clip': 3.0, 'log_increment': 50, 'subtract_background': None, 'use_source_posn': None, 'center_xy': None, 'apply_apcorr': True, 'soss_threshold': 0.01, 'soss_n_os': 2, 'soss_transform': None, 'soss_tikfac': None, 'soss_width': 40.0, 'soss_bad_pix': 'model', 'soss_modelname': None}
2022-10-05 23:10:29,693 - stpipe.Spec2Pipeline.extract_1d - INFO - Step skipped.
2022-10-05 23:10:29,696 - stpipe.Spec2Pipeline.extract_1d - INFO - Step extract_1d done
2022-10-05 23:10:29,696 - stpipe.Spec2Pipeline - INFO - Finished processing product ifu_g395h_f290lp_nrs2_uncal
2022-10-05 23:10:29,697 - stpipe.Spec2Pipeline - INFO - Ending calwebb_spec2
2022-10-05 23:10:29,697 - stpipe.Spec2Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-05 23:10:29,698 - stpipe.Spec2Pipeline - INFO - Step Spec2Pipeline done
Running flat field test for IFU...
flat_field_file --> Grating:G395H Filter:F290LP LAMP:REF
Getting and reading the D-, S-, and F-flats for this specific IFU configuration...
Using D-flat: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_dflat_nrs2_f_01.03.fits
Using S-flat: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_IFU_sflat_G395H_OPAQUE_FLAT3_nrs2_f_01.01.fits
Using F-flat: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_IFU_fflat_F290LP_01.01.fits
Now looping through the slices, this may take some time...
Working with slice: 00
Subwindow origin: px0=1 py0=782
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 00
Absolute Flat Difference : mean = 2.168e-08 median = 6.346e-08 stdev = 6.939e-05
Maximum AbsoluteFlat Difference = 8.200e-04
Minimum AbsoluteFlat Difference = -6.043e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 01
Subwindow origin: px0=1 py0=1198
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 01
Absolute Flat Difference : mean = -6.246e-08 median = -3.189e-08 stdev = 6.755e-05
Maximum AbsoluteFlat Difference = 8.549e-04
Minimum AbsoluteFlat Difference = -8.739e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 02
Subwindow origin: px0=1 py0=732
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 02
Absolute Flat Difference : mean = 1.348e-08 median = 3.126e-08 stdev = 6.590e-05
Maximum AbsoluteFlat Difference = 8.870e-04
Minimum AbsoluteFlat Difference = -6.725e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 03
Subwindow origin: px0=1 py0=1248
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 03
Absolute Flat Difference : mean = -3.159e-08 median = 9.957e-08 stdev = 6.616e-05
Maximum AbsoluteFlat Difference = 9.555e-04
Minimum AbsoluteFlat Difference = -8.231e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 04
Subwindow origin: px0=1 py0=683
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 04
Absolute Flat Difference : mean = -1.194e-07 median = 3.553e-09 stdev = 6.698e-05
Maximum AbsoluteFlat Difference = 7.617e-04
Minimum AbsoluteFlat Difference = -7.885e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 05
Subwindow origin: px0=1 py0=1297
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 05
Absolute Flat Difference : mean = -5.244e-08 median = 5.983e-08 stdev = 6.545e-05
Maximum AbsoluteFlat Difference = 8.285e-04
Minimum AbsoluteFlat Difference = -8.851e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 06
Subwindow origin: px0=1 py0=634
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 06
Absolute Flat Difference : mean = 6.762e-08 median = 1.477e-08 stdev = 6.570e-05
Maximum AbsoluteFlat Difference = 8.919e-04
Minimum AbsoluteFlat Difference = -8.686e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 07
Subwindow origin: px0=1 py0=1346
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 07
Absolute Flat Difference : mean = -8.432e-08 median = 6.254e-08 stdev = 6.567e-05
Maximum AbsoluteFlat Difference = 8.171e-04
Minimum AbsoluteFlat Difference = -7.579e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 08
Subwindow origin: px0=1 py0=585
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 08
Absolute Flat Difference : mean = 7.427e-08 median = -8.539e-08 stdev = 6.572e-05
Maximum AbsoluteFlat Difference = 8.631e-04
Minimum AbsoluteFlat Difference = -8.838e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 09
Subwindow origin: px0=1 py0=1395
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 09
Absolute Flat Difference : mean = -6.399e-08 median = 1.396e-07 stdev = 6.595e-05
Maximum AbsoluteFlat Difference = 8.630e-04
Minimum AbsoluteFlat Difference = -8.051e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 10
Subwindow origin: px0=1 py0=535
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 10
Absolute Flat Difference : mean = 3.099e-08 median = 1.684e-08 stdev = 6.553e-05
Maximum AbsoluteFlat Difference = 8.349e-04
Minimum AbsoluteFlat Difference = -9.347e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 11
Subwindow origin: px0=1 py0=1444
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 11
Absolute Flat Difference : mean = -1.799e-07 median = -2.454e-07 stdev = 6.592e-05
Maximum AbsoluteFlat Difference = 8.642e-04
Minimum AbsoluteFlat Difference = -9.221e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 12
Subwindow origin: px0=1 py0=486
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 12
Absolute Flat Difference : mean = -5.865e-08 median = 6.309e-08 stdev = 6.552e-05
Maximum AbsoluteFlat Difference = 8.789e-04
Minimum AbsoluteFlat Difference = -1.459e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 13
Subwindow origin: px0=1 py0=1492
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 13
Absolute Flat Difference : mean = -2.238e-08 median = 1.063e-07 stdev = 6.566e-05
Maximum AbsoluteFlat Difference = 8.500e-04
Minimum AbsoluteFlat Difference = -8.229e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 14
Subwindow origin: px0=1 py0=437
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 14
Absolute Flat Difference : mean = 1.212e-07 median = 1.695e-07 stdev = 6.529e-05
Maximum AbsoluteFlat Difference = 9.252e-04
Minimum AbsoluteFlat Difference = -8.729e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 15
Subwindow origin: px0=1 py0=1541
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 15
Absolute Flat Difference : mean = -8.724e-08 median = 4.949e-08 stdev = 6.587e-05
Maximum AbsoluteFlat Difference = 7.416e-04
Minimum AbsoluteFlat Difference = -8.280e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 16
Subwindow origin: px0=1 py0=388
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 16
Absolute Flat Difference : mean = 7.301e-09 median = 1.442e-07 stdev = 6.518e-05
Maximum AbsoluteFlat Difference = 9.374e-04
Minimum AbsoluteFlat Difference = -9.005e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 17
Subwindow origin: px0=1 py0=1590
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 17
Absolute Flat Difference : mean = 2.076e-07 median = -1.157e-09 stdev = 6.630e-05
Maximum AbsoluteFlat Difference = 7.847e-04
Minimum AbsoluteFlat Difference = -7.677e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 18
Subwindow origin: px0=1 py0=338
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 18
Absolute Flat Difference : mean = -1.832e-07 median = 7.087e-08 stdev = 6.453e-05
Maximum AbsoluteFlat Difference = 8.030e-04
Minimum AbsoluteFlat Difference = -9.155e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 19
Subwindow origin: px0=1 py0=1639
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 19
Absolute Flat Difference : mean = -2.059e-07 median = -1.496e-08 stdev = 6.755e-05
Maximum AbsoluteFlat Difference = 7.233e-04
Minimum AbsoluteFlat Difference = -9.334e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 20
Subwindow origin: px0=1 py0=289
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 20
Absolute Flat Difference : mean = 1.760e-08 median = 2.096e-07 stdev = 6.376e-05
Maximum AbsoluteFlat Difference = 8.165e-04
Minimum AbsoluteFlat Difference = -8.683e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 21
Subwindow origin: px0=1 py0=1688
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 21
Absolute Flat Difference : mean = 2.248e-08 median = -4.151e-09 stdev = 6.668e-05
Maximum AbsoluteFlat Difference = 8.358e-04
Minimum AbsoluteFlat Difference = -8.637e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 22
Subwindow origin: px0=1 py0=240
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 22
Absolute Flat Difference : mean = -1.710e-07 median = 9.461e-08 stdev = 6.571e-05
Maximum AbsoluteFlat Difference = 7.390e-04
Minimum AbsoluteFlat Difference = -9.306e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 23
Subwindow origin: px0=1 py0=1737
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 23
Absolute Flat Difference : mean = 2.661e-07 median = 1.713e-07 stdev = 6.725e-05
Maximum AbsoluteFlat Difference = 8.318e-04
Minimum AbsoluteFlat Difference = -8.463e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 24
Subwindow origin: px0=1 py0=190
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 24
Absolute Flat Difference : mean = -2.886e-09 median = 1.421e-07 stdev = 6.573e-05
Maximum AbsoluteFlat Difference = 8.327e-04
Minimum AbsoluteFlat Difference = -8.911e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 25
Subwindow origin: px0=1 py0=1785
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 25
Absolute Flat Difference : mean = -7.253e-08 median = 1.034e-07 stdev = 6.775e-05
Maximum AbsoluteFlat Difference = 8.188e-04
Minimum AbsoluteFlat Difference = -8.996e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 26
Subwindow origin: px0=1 py0=141
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 26
Absolute Flat Difference : mean = 1.225e-07 median = 3.557e-08 stdev = 6.578e-05
Maximum AbsoluteFlat Difference = 8.011e-04
Minimum AbsoluteFlat Difference = -8.995e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 10%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 27
Subwindow origin: px0=1 py0=1834
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 27
Absolute Flat Difference : mean = -7.991e-08 median = 1.957e-07 stdev = 7.064e-05
Maximum AbsoluteFlat Difference = 1.263e-03
Minimum AbsoluteFlat Difference = -1.566e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 28
Subwindow origin: px0=1 py0=91
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 28
Absolute Flat Difference : mean = -7.503e-08 median = 1.017e-07 stdev = 5.715e-05
Maximum AbsoluteFlat Difference = 6.428e-04
Minimum AbsoluteFlat Difference = -7.690e-04
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 7%
-> 3xtheshold = 0%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
Working with slice: 29
Subwindow origin: px0=1 py0=1883
Looping through the wavelength, this may take a little time ...
Flat value differences for slice number: 29
Absolute Flat Difference : mean = 3.974e-07 median = 2.696e-07 stdev = 9.526e-05
Maximum AbsoluteFlat Difference = 1.397e-02
Minimum AbsoluteFlat Difference = -1.484e-03
Percentage of pixels where median of absolute differences is greater than:
-> 1xtheshold = 11%
-> 3xtheshold = 1%
-> 5xtheshold = 0%
Making the plot for this slice...
*** Result of the test: PASSED
*** Final result for flat_field test will be reported as PASSED ***
('* Script flattest_ifu.py script took ', '37.751657907168074 minutes to finish.')
Did flat_field for ifu_g395h_f290lp validation test passed? All slices PASSED flat_field test.
Closing file: popenfile(path='/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/ifu_g395h_f290lp_nrs2_interpolatedflat.fits', fd=57, position=16804800, mode='r', flags=557056)
Closing file: popenfile(path='/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_dflat_nrs2_f_01.03.fits', fd=58, position=680480640, mode='r', flags=557056)
Closing file: popenfile(path='/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_IFU_sflat_G395H_OPAQUE_FLAT3_nrs2_f_01.01.fits', fd=59, position=50500800, mode='r', flags=557056)
Closing file: popenfile(path='/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmp_31lz4w5/nirspec_IFU_fflat_F290LP_01.01.fits', fd=69, position=86400, mode='r', flags=557056)
# Quickly see if the test passed
print('These are the final results of the tests: ')
for key, val in results_dict.items():
if not isinstance(val, str):
if val:
val = 'PASSED'
else:
val = 'FAILED'
print('{:<42} {:<8}'.format(key, val))
These are the final results of the tests: ifu_g395h_f290lp_nrs1_uncal.fits PASSED ifu_g395h_f290lp_nrs2_uncal.fits PASSED
Author: Maria A. Pena-Guerrero, Sr. Science Software Engineer, NIRSpec
Updated On: Sep/23/2022